Techniques for medical image retrieval

ABSTRACT

A technique for searching for an image includes calculating wavelet features of a plurality of images. A keyword included in radiographic interpretation information is extracted for each of the stored images. The calculated wavelet features and the extracted keywords are stored in association with the respective stored images. A newly taken image is acquired and a wavelet feature of the acquired image is calculated. A keyword included in radiographic interpretation information corresponding to the acquired image is extracted and a search for similar radiographic interpretation information from the stored keywords is performed. A wavelet feature-based spatial distance between the acquired image and each of images corresponding to the radiographic interpretation information found is calculated. A search result of any images for which the calculated wavelet feature-based spatial distance is shorter than a predetermined value is output, in ascending order of the calculated wavelet feature-based spatial distance.

This application is a national stage of International Application No.PCT/JP2012/074063, entitled “MEDICAL-IMAGE RETRIEVAL METHOD, DEVICE, ANDCOMPUTER PROGRAM,” filed Sep. 20, 2012, which claims priority toJapanese Patent Application No. 2011-272174, filed Dec. 13, 2011. Thedisclosure of International Application No. PCT/JP2012/074063 is herebyincorporated herein by reference in its entirety for all purposes.

BACKGROUND

The disclosure relates to image retrieval and, more specifically, tomedical image retrieval of an image having high similarity to anacquired image from among a plurality of stored images.

In medical practice, it is important to recognize the internal states ofa patient on the basis of images acquired by radiography. An X-ray imageof a patient can be compared with X-ray images taken in the past toidentify the causes of symptoms of the patient. This enables selectionof appropriate medical treatment, leading to early improvement of thesymptoms of the patient.

Japanese Unexamined Patent Publication No. 2005-237781 discloses acardiomagnetometer which measures a cardiac magnetic signal and comparesthe measured cardiac magnetic signal with those measured in the past todetermine whether the person is suffering from a cardiac disease orwhether he/she is a candidate thereof. Japanese Unexamined PatentPublication No. 2008-077163 discloses a search system that calculates asimilarity between a latest diagnosis image and each of diagnosis imagesincluded in past interpretation report data and, in considerationtogether with the rate of occurrence of a diagnosis result name in thepast interpretation report data, searches for a past diagnosis image andthe interpretation report data corresponding thereto.

In Japanese Unexamined Patent Publication No. 2005-237781, thedetermination is made on the basis of a cardiac magnetic signal, whichis indirect information, and therefore an occurrence of a cardiacdisease cannot be distinguished from an occurrence of any other diseaseindicated by a similar signal. In Japanese Unexamined Patent PublicationNo. 2008-077163, although the similarity between images is calculated byusing a Euclidean distance between feature parts in the images, it isnecessary to determine which part of the picked-up image is the featurepart and, as such, the determination might depend on a doctor's skill inimage reading.

BRIEF SUMMARY

A technique for searching for an image similar to a newly taken image,from among medical images based on past cases includes calculatingwavelet features of a plurality of images that have been taken andstored in the past. A keyword included in radiographic interpretationinformation is extracted for each stored image. The calculated waveletfeatures and the extracted keywords are stored in association with therespective stored images. A newly taken image is acquired and a waveletfeature of the acquired image is calculated. A keyword included inradiographic interpretation information corresponding to the acquiredimage is extracted and, on the basis of the extracted keyword, a searchfor similar radiographic interpretation information from the storedkeywords is performed. A wavelet feature-based spatial distance betweenthe acquired image and each of images corresponding to the radiographicinterpretation information found is calculated. A search result of anyimages for which the calculated wavelet feature-based spatial distanceis shorter than a predetermined value is output, in ascending order ofthe calculated wavelet feature-based spatial distance.

The above summary contains simplifications, generalizations andomissions of detail and is not intended as a comprehensive descriptionof the claimed subject matter but, rather, is intended to provide abrief overview of some of the functionality associated therewith. Othersystems, methods, functionality, features and advantages of the claimedsubject matter will be or will become apparent to one with skill in theart upon examination of the following figures and detailed writtendescription.

The above as well as additional objectives, features, and advantages ofthe present invention will become apparent in the following detailedwritten description.

BRIEF DESCRIPTION OF THE DRAWINGS

The description of the illustrative embodiments is to be read inconjunction with the accompanying drawings, wherein:

FIG. 1 is a block diagram schematically showing the configuration of amedical image search apparatus according to an embodiment of the presentdisclosure;

FIG. 2 is a functional block diagram of the medical image searchapparatus according to an embodiment of the present disclosure;

FIG. 3 illustrates coordinate setting within an image used in themedical image search apparatus according to an embodiment of the presentdisclosure;

FIGS. 4A and 4B show, by way of example, a two-dimensional Gabor waveletfunction;

FIG. 5 is a schematic diagram showing the directions of thetwo-dimensional Gabor wavelet function used in a medical image searchapparatus according to an embodiment of the present disclosure;

FIG. 6 shows, by way of example, the data structure of visual wordsstored in a visual word storage unit in a medical image search apparatusaccording to an embodiment of the present disclosure;

FIG. 7 shows, by way of example, keyword extraction by a medical imagesearch apparatus according to an embodiment of the present disclosure;

FIG. 8 shows, by way of example, a histogram according to an embodimentof the present disclosure;

FIG. 9 shows an example of a search result display screen used in amedical image search apparatus according to an embodiment of the presentdisclosure; and

FIG. 10 is a flowchart illustrating a processing procedure of a CPU in amedical image search apparatus according to an embodiment of the presentdisclosure.

DETAILED DESCRIPTION

The illustrative embodiments provide a method, an apparatus, and acomputer program product for retrieval of an image having highsimilarity to an acquired image, from among a plurality of storedimages.

In the following detailed description of exemplary embodiments of theinvention, specific exemplary embodiments in which the invention may bepracticed are described in sufficient detail to enable those skilled inthe art to practice the invention, and it is to be understood that otherembodiments may be utilized and that logical, architectural,programmatic, mechanical, electrical and other changes may be madewithout departing from the spirit or scope of the present invention. Thefollowing detailed description is, therefore, not to be taken in alimiting sense, and the scope of the present invention is defined by theappended claims and equivalents thereof.

It should be understood that the use of specific component, device,and/or parameter names are for example only and not meant to imply anylimitations on the invention. The invention may thus be implemented withdifferent nomenclature/terminology utilized to describe thecomponents/devices/parameters herein, without limitation. Each termutilized herein is to be given its broadest interpretation given thecontext in which that term is utilized. As may be used herein, the term‘coupled’ may encompass a direct connection between components orelements or an indirect connection between components or elementsutilizing one or more intervening components or elements.

Embodiments of the present disclosure are provided in view of the abovebackground and provide a medical image search method, apparatus, andcomputer program capable of using a newly taken X-ray image to searchwith high accuracy for an existing image corresponding to a similarcase.

A disclosed method, e.g., executed by a medical image search apparatus,searches for an image similar to a newly taken image from among medicalimages based on past cases. The method may include: calculating waveletfeatures of a plurality of images that have been taken and stored in thepast; extracting a keyword included in the radiographic interpretationinformation for each stored image; storing the calculated waveletfeatures and the extracted keywords in association with the respectivestored images; acquiring a newly taken image; calculating a waveletfeature of the acquired image; extracting a keyword included inradiographic interpretation information corresponding to the acquiredimage and, on the basis of the extracted keyword, searching for similarradiographic interpretation information from the stored keywords;calculating a wavelet feature-based spatial distance between theacquired image and each of images corresponding to the radiographicinterpretation information found; and outputting as a search result anyimages the calculated spatial distance of which is shorter than apredetermined value, in ascending order of the spatial distance.

The method may include calculating a wavelet feature as atwo-dimensional Gabor wavelet feature.

The method may further include calculating frequency distributionvectors for all images, by calculating M said wavelet features (M is anatural number of 2 or greater) for each image and binarizing therespective wavelet features for conversion into an M-dimensional bitstring. The spatial distance may be calculated as an angle between thecalculated frequency distribution vectors.

An apparatus, e.g., a medical image search apparatus, for searching foran image similar to a newly taken image, from among medical images basedon past cases, may include: a feature calculation unit for calculatingwavelet features of a plurality of images that have been taken andstored in the past; a keyword extraction unit for extracting a keywordincluded in radiographic interpretation information for each storedimage; an information storage unit for storing the calculated waveletfeatures and the extracted keywords in association with the respectivestored images; an image acquisition unit for acquiring a newly takenimage; a wavelet feature calculation units for calculating a waveletfeature of the acquired image; a radiographic interpretation informationsearch unit for extracting a keyword included in radiographicinterpretation information corresponding to the acquired image and, onthe basis of the extracted keyword, searching for similar radiographicinterpretation information from the stored keywords; a spatial distancecalculation unit for calculating a wavelet feature-based spatialdistance between the acquired image and each of images corresponding tothe radiographic interpretation information found; and an output unitfor outputting as a search result any images the calculated spatialdistance of which is shorter than a predetermined value, in ascendingorder of the spatial distance.

The feature calculation unit and the wavelet feature calculation unitmay each calculate a two-dimensional Gabor wavelet feature as thewavelet feature.

The apparatus may further include a frequency distribution vectorcalculation unit for calculating frequency distribution vectors for allimages, by calculating M said wavelet features (M is a natural number of2 or greater) for each image and binarizing the respective waveletfeatures for conversion into an M-dimensional bit string, and thespatial distance calculation unit may calculate the spatial distance asan angle between the calculated frequency distribution vectors.

A computer program according, e.g., executable by a medical image searchapparatus, may search for an image similar to a newly taken image, fromamong medical images based on past cases. The program causes anapparatus to function as: a feature calculation unit for calculatingwavelet features of a plurality of images that have been taken andstored in the past; a keyword extraction unit for extracting a keywordincluded in radiographic interpretation information for each storedimage; an information storage unit for storing the calculated waveletfeatures and the extracted keywords in association with the respectivestored images; an image acquisition unit for acquiring a newly takenimage; a wavelet feature calculation unit for calculating a waveletfeature of the acquired image; an interpretation information search unitfor extracting a keyword included in radiographic interpretationinformation corresponding to the acquired image and, on the basis of theextracted keyword, searching for similar radiographic interpretationinformation from the stored keywords; a spatial distance calculationunit for calculating a wavelet feature-based spatial distance betweenthe acquired image and each of images corresponding to the radiographicinterpretation information found; and an output unit for outputting as asearch result any images the calculated spatial distance of which isshorter than a predetermined value, in ascending order of the spatialdistance.

The computer program according may cause the feature calculation unitand the wavelet feature calculation unit to calculate a two-dimensionalGabor wavelet feature as the wavelet feature.

The computer program may further causes the apparatus to function as afrequency distribution vector calculation unit for calculating frequencydistribution vectors for all images, by calculating M said waveletfeatures (M is a natural number of 2 or greater) for each image andbinarizing the respective wavelet features for conversion into anM-dimensional bit string, and the cause the spatial distance calculationunit to calculate the spatial distance as an angle between thecalculated frequency distribution vectors.

According to the present disclosure, a wavelet feature indicating thefeature of an acquired medical image can be used to search for a similarimage from among the images stored as past cases. Even an inexperienceddoctor is able to find an image corresponding to the most similar caseand, thus, to select appropriate medical treatment.

A medical image search apparatus for searching for an image similar to anewly taken image, from among medical images based on past casesaccording to an embodiment of the present disclosure is specificallydescribed below with reference to the drawings. The followingembodiments do not restrict the claimed invention, and all thecombinations of the features described in the embodiment are notnecessarily indispensable.

Further, the present invention can be carried out in many differentmodes, and should not be understood only from the description given forthe embodiment. Through the whole description of the embodiment, thesame elements are denoted by the same reference numerals.

While an apparatus comprising a computer system having a computerprogram introduced therein will be described in the followingembodiment, it should be apparent to those skilled in the art that partof the present invention may be implemented as a computer-executablecomputer program. Therefore, the present invention can take the form ofan embodiment as hardware, e.g., a medical image search apparatus, thatsearches for an image similar to a newly taken image from among medicalimages based on past cases or an embodiment as a combination of softwareand hardware. The computer program may be recorded on an arbitrarycomputer-readable recording medium such as a hard disk, a DVD, a CD, anoptical storage device, or a magnetic storage device.

According to an embodiment, a wavelet feature indicating the feature ofan acquired medical image can be used to search for an image similar tothe acquired image from among the images stored as past cases. Even aninexperienced doctor is able to find an image corresponding to the mostsimilar case and, thus, to select appropriate medical treatment.

FIG. 1 is a block diagram schematically showing the configuration of amedical image search apparatus according to an embodiment. The medicalimage search apparatus 1 according to the embodiment at least includes:a central processing unit (CPU) or processor 11, a memory 12, a storagedevice 13, an I/O interface 14, a video interface 15, a portable diskdrive 16, a communication interface 17, and an internal bus 18 forconnecting the above-described hardware components.

The CPU 11 is connected via the internal bus 18 to the hardwarecomponents of the medical image search apparatus 1 as described above.The CPU 11 controls the operations of those hardware components, andalso executes various software functions in accordance with a computerprogram 100 stored in the storage device 13. The memory 12 is made up ofa volatile memory such as an SRAM or an SDRAM, in which a load module isdeployed at the time of execution of the computer program 100. Temporarydata generated during the execution of the computer program 100 is alsostored in the memory 12.

The storage device 13 includes a built-in fixed storage (hard disk), aROM, and others. The computer program 100 stored in the storage device13 is one that has been downloaded by the portable disk drive 16 from aportable recording medium 90 such as a DVD or a CD-ROM that recordsinformation such as data and programs. At run-time, the computer program100 is deployed from the storage device 13 to the memory 12 forexecution. The computer program 100 may of course be downloaded from anexternal computer connected via the communication interface 17.

The storage device 13 includes a medical image storage unit 131, aradiographic interpretation information storage unit 132, a visual wordstorage unit 133, and a frequency distribution information storage unit134. The medical image storage unit 131 stores image data of X-rayimages taken in the past. The unit 131 stores the image data inassociation with identification information for identifying radiographicinterpretation information.

The radiographic interpretation information storage unit 132 storesresults of diagnoses that doctors have made by interpreting medicalimages taken in the past. For example, a doctor's diagnosis such as“nodular shadow found in left lung field, upper lobe; squamous cellcarcinoma suspected; workup by HR-CT instructed” is stored in the formof text data in association with identification information.

The visual word storage unit 133 stores, as visual words, Gabor waveletfeatures which will be described later. The frequency distributioninformation storage unit 134 stores frequency distribution vectors ofvalues obtained by binarizing calculated wavelet features and convertingthem into M-dimensional bit strings.

The communication interface 17 is connected to the internal bus 18, andto an external network such as the Internet, a LAN, or a WAN, so that itis able to transmit data to and receive data from an external computerand so on.

The I/O interface 14 is connected to input devices such as a keyboard 21and a mouse 22, and accepts input of data. The video interface 15 isconnected to a display device 23 such as a CRT display or a liquidcrystal display, and displays a detected result on the display device23.

FIG. 2 is a functional block diagram of the medical image searchapparatus 1 according to an embodiment. In FIG. 2, a feature calculationunit 201 in the medical image search apparatus 1 calculates waveletfeatures of a plurality of images taken and stored in the past. In anembodiment, Gabor wavelet features are calculated as the waveletfeatures.

FIG. 3 illustrates coordinate setting within an image used in themedical image search apparatus 1 according to an embodiment. As shown inFIG. 3, an image with an origin at an upper left corner thereof andhaving m pixels in an x direction and n pixels in a y direction isdefined as s(x, y). The coordinates of an i-th pixel P, (i is a naturalnumber) are represented as P_(i)(x_(i), y_(i)).

First, the coordinates P_(i)(x_(i), y_(i)) are affine-transformed tocoordinates (X_(i), Y_(i)) in accordance with the following expression(1).[X_(i),Y_(i),1]=[x_(i),y_(i)1]A  (1)

In the above expression (1), the matrix A is a 3×3 affine transformationmatrix. The affine transformation to shift the entire image by tx in thex direction and by ty in the y direction can be expressed by thefollowing expression (2), and the affine transformation to rotate theentire image by an angle θ can be expressed by the following expression(3).

$\begin{matrix}{\left\lbrack {X_{i},Y_{i},1} \right\rbrack = {\left\lbrack {x_{i},y_{i},1} \right\rbrack\begin{bmatrix}1 & 0 & 0 \\0 & 1 & 0 \\{tx} & {ty} & 1\end{bmatrix}}} & (2) \\{\left\lbrack {X_{i},Y_{i},1} \right\rbrack = {\left\lbrack {x_{i},y_{i},1} \right\rbrack\begin{bmatrix}{\cos\;\theta} & {\sin\;\theta} & 0 \\{{- \sin}\;\theta} & {\cos\;\theta} & 0 \\0 & 0 & 1\end{bmatrix}}} & (3)\end{matrix}$

A two-dimensional Gabor wavelet function is defined, with respect to thecoordinate values (x with dot, y with dot) after the affinetransformation for rotation, as in the following expression (4).

$\begin{matrix}{{\psi_{r}\left( {x,y} \right)} = {{{{g_{\sigma}\left( {\overset{.}{x},\overset{.}{y}} \right)}\left\lbrack {{\mathbb{e}}^{{\mathbb{i}}\; u_{0}\overset{.}{x}} - {\mathbb{e}}^{- {({u_{0}\sigma})}^{2}}} \right\rbrack}\begin{bmatrix}\overset{.}{x} \\\overset{.}{y}\end{bmatrix}} = {\begin{bmatrix}{\cos\;\theta_{r}} & {\sin\;\theta_{r}} \\{{- \sin}\;\theta_{r}} & {\cos\;\theta_{r}}\end{bmatrix}\begin{bmatrix}x \\y\end{bmatrix}}}} & (4)\end{matrix}$

The two-dimensional Gabor wavelet function is composed of a real partand an imaginary part. FIGS. 4A and 4B show an example of atwo-dimensional Gabor wavelet function. Specifically, FIGS. 4A and 4Bshow examples of the real part and imaginary part, respectively, of thetwo-dimensional Gabor wavelet function. As seen from FIGS. 4A and 4B,the real part of the two-dimensional Gabor wavelet function has ahat-like wavy form with its maximum value located near (x, Y)=(0, 0). Inthe above expression (4), u₀ represents the frequency of that waveform,and σ represents the width of that hat shape. Further, r represents thedirection, which will be described later.

The window function g_(σ) in the above expression (4) is atwo-dimensional Gaussian function, which can be expressed by thefollowing expression (5).

$\begin{matrix}{{g_{\sigma}\left( {\overset{.}{x},\overset{.}{y}} \right)} = {\frac{1}{4\;\pi\;\sigma}{\mathbb{e}}^{\frac{- 1}{4\;\sigma^{2}}{({{\overset{.}{x}}^{2} + {\overset{.}{y}}^{2}})}}}} & (5)\end{matrix}$

Using the two-dimensional Gabor wavelet function, the Gabor waveletfeatures for an acquired image s(x, y) can be calculated by thefollowing expression (6). The lattice point at which the absolute valueof the Gabor wavelet feature has a maximum value and the Gabor waveletfeatures in the vicinity of that lattice point are invariant even whenthe image is subjected to affine transformation such as scaling,rotation, etc., so that they are suitably used as the feature values ofan image.

$\begin{matrix}{{G_{j,r}\left( {x_{0},y_{0}} \right)} = {a^{- j}{\int{\int{{s\left( {x,y} \right)}{\psi_{r}\left( {\frac{x - x_{0}}{a^{j}},\frac{y - y_{0}}{a^{j}}} \right)}{\mathbb{d}x}{\mathbb{d}y}}}}}} & (6)\end{matrix}$

In the above expression (6), a^(j) and a^(−j) are parameters indicatingthe degrees of dilation (scaling), and x₀ and y₀ represent shift.Further, r represents the direction. In the present embodiment, theGabor wavelet features in eight directions are calculated.

FIG. 5 is a schematic diagram showing the directions of thetwo-dimensional Gabor wavelet function used in the medical image searchapparatus 1 according to an embodiment. As shown in FIG. 5, the Gaborwavelet features are calculated in directions (1) to (8), i.e. in eightdirections spaced every 22.5 degrees from a prescribed direction.

The calculation of the Gabor wavelet features makes it possible tocalculate the wavelet feature values that accommodate or absorbvariations in shape of the human organs, for example, thereby enabling ahigh-precision search for a similar image.

For example, in the case where the above expression (6) is used tocalculate the Gabor wavelet features for each coordinate point (x, y)(lattice point within an image), eight directions (r=1 to 8) and fivescales (j=1 to 5) are selected to calculate 40 Gabor wavelet featuresfor one coordinate point. Here, the scales 1 to 5 indicate the levels ofenlargement/reduction. For example, a greater value indicates a greaterdegree of enlargement. From the Gabor wavelet features calculated, thosehaving the absolute values of not less than a predetermined thresholdvalue are extracted, and the Gabor wavelet feature having a maximumvalue among them is selected.

The fact that the absolute value of the Gabor wavelet feature takes amaximum value means that the absolute value of the integral in the aboveexpression (6) is maximum. The feature value remains unchanged even whenthe average brightness of the image is changed, the scale of the imageis changed, or the image is rotated.

In the present embodiment, the Gabor wavelet features in eightdirections in the scale where a maximum value is obtained, as well asthe Gabor wavelet features in the eight directions in each of thepreceding and succeeding scales, namely 24 (3 scales×8 directions) Gaborwavelet features in total, are stored as a set of visual words in thevisual word storage unit 133.

FIG. 6 shows, by way of example, the data structure of visual wordsstored in the visual word storage unit 133 in the storage device 13 inthe medical image search apparatus 1 according to an embodiment. Asshown in FIG. 6, 24 Gabor wavelet features which have been calculatedare listed and stored corresponding to each identification number 1, 2,3, . . . . More specifically, “1” at the beginning is the identificationnumber, which is followed by a blank space, and the numerical valuesfollowing “1:” to “24:” are the 24 Gabor wavelet features calculated.FIG. 6 shows the visual words in the case where there are three maximumvalues within one image. Thus, in FIG. 6, the visual words are storedcorresponding to three identification numbers “1”, “2”, and “3”. Whenthere is one maximum value, there is naturally only one identificationnumber “1”.

Returning to FIG. 2, a keyword extraction unit 202 extracts keywordsthat are included in the radiographic interpretation information storedin the radiographic interpretation information storage unit 132 in thestorage device 13 corresponding to the past images stored in the medicalimage storage unit 131 in the storage device 13. For example, in thecase where radiographic interpretation information reading: “nodularshadow found in left lung field, upper lobe; squamous cell carcinomasuspected; workup by HR-CT instructed” is stored in the radiographicinterpretation information storage unit 132 in the storage device 13,syntax analysis is carried out using morphological analysis or the liketo extract keywords, which are classified as “site”, “symptom”, “diseasename”, “action”, etc.

FIG. 7 shows, by way of example, keyword extraction by the medical imagesearch apparatus 1 according to one embodiment. In the example shown inFIG. 7, through the syntactic analysis of “nodular shadow found in leftlung field, upper lobe; squamous cell carcinoma suspected; workup byHR-CT instructed”, the following keywords have been extracted: “leftlung field, upper lobe” as “site”, “nodular shadow” as “symptom”,“squamous cell carcinoma suspected” as “disease name”, and “workup byHR-CT” as “action”.

An information storage unit 203 stores the wavelet features calculatedin the above-described manner and the extracted keywords, as visualwords, in the visual word storage unit 133 in the storage device 13. Theunit 203 stores the wavelet features and the keywords in associationwith the past images stored in the medical image storage unit 131 in thestorage device 13.

An image acquisition unit 204 acquires a newly taken image. The imageacquisition unit 204 preferably acquires the radiographic interpretationinformation corresponding thereto at the same time. This can helpnarrowing down the images to be searched for a similar image.

A wavelet feature calculation unit 205 calculates wavelet features ofthe acquired image, similarly as in the above-described manner, andstores the calculated features as visual words in the visual wordstorage unit 133.

A radiographic interpretation information search unit 206 extracts atleast one keyword included in the radiographic interpretationinformation for the acquired image and, on the basis of the extractedkeyword, searches for similar radiographic interpretation informationfrom the keywords stored in the radiographic interpretation informationstorage unit 132 in the storage device 13. In this manner, it ispossible to effectively narrow down the images to be searched for asimilar image.

A spatial distance calculation unit 207 calculates a waveletfeature-based spatial distance between the acquired image and an imagecorresponding to the radiographic interpretation information found. Morespecifically, for each pixel, M wavelet features (M is a natural numberof 2 or greater), for example 24 wavelet features, are calculated, whichare then binarized and converted to an M-dimensional bit string (M=24).

A frequency distribution vector calculation unit 209 generates ahistogram indicating the frequency distribution of the values of the24-dimensional bit strings obtained through conversion. Such a histogramis generated, not only for a newly acquired image, but also for all theimages stored in the medical image storage unit 131, or for the imagescorresponding to the radiographic interpretation information found bythe radiographic interpretation information search unit 206.

FIG. 8 shows an example of a histogram according to one embodiment. Inthis example, 2²⁴ values are taken along the horizontal axis, andfrequency distribution is obtained for the respective values. Thefrequency distribution for each image can be used as a frequencydistribution vector, to effectively narrow down the images to besearched for a similar image. The information regarding the generatedhistograms is stored in the frequency distribution information storageunit 134 in the storage device 13.

The spatial distance calculation unit 207 calculates the spatialdistance between the newly acquired image and an image stored in themedical image storage unit 131, as an angle between the calculatedfrequency distribution vectors. More specifically, when the frequencydistribution vector of the newly acquired image is represented as V₁ andthe frequency distribution vector of an image stored in the medicalimage storage unit 131 is represented as V₂, then the spatial distanceis calculated as the cosine of the angle φ between the two vectors, i.e.cos φ, in accordance with the following expression (7).

$\begin{matrix}{{\cos\;\phi} = \frac{\left\langle {V_{1},V_{2}} \right\rangle}{{V_{1}} \cdot {V_{2}}}} & (7)\end{matrix}$

In the above expression (7), <V₁, V₂> indicates the inner product of thevectors V₁ and V₂, and the denominator indicates the product between thenorm (length) of the vector V₁ and the norm of the vector V₂.

Returning to FIG. 2, a result output unit (output unit) 208 outputs, asa search result, the images having the calculated spatial distancesshorter than a predetermined value, in ascending order of their spatialdistance. It can be determined that an image having a shorter spatialdistance has a higher similarity to the newly acquired image. It is thuspossible to select appropriate medical treatment by referring to thesimilar images taken in the past.

It is noted that feature vectors may be overlaid on an image beingdisplayed on the display device 23 as a search result. FIG. 9 shows anexample of a search result display screen used in the medical imagesearch apparatus 1 according to one embodiment.

As shown in FIG. 9, a past image that has been determined to be mostsimilar to the acquired image is displayed and, of the wavelet features,those greater than a predetermined value are displayed overlaid on theimage as the feature vectors. The length of each arrow indicates themagnitude of the feature value. The direction of each arrow indicatesthe one of the eight directions in which the feature value is greatest.The scales may also be distinguished by colors, line types, and so on.

FIG. 10 is a flowchart illustrating the processing procedure of the CPU11 in the medical image search apparatus 1 according to an embodiment.Referring to FIG. 10, the CPU 11 in the medical image search apparatus 1calculates wavelet features of a plurality of images that have beentaken and stored in the past (S1001). In the present embodiment, Gaborwavelet features are calculated as the wavelet features.

The CPU 11 extracts keywords included in the radiographic interpretationinformation for each of the stored past images (S1002), and stores theextracted keywords and the wavelet features calculated in theabove-described manner, in association with the stored past images.

The CPU 11 acquires a newly taken image, in association withradiographic interpretation information (S1003), calculates waveletfeatures of the acquired image in the above-described manner (S1004),and stores the calculated wavelet features as visual words in the visualword storage unit 133.

The CPU 11 extracts at least one keyword included in the radiographicinterpretation information for that acquired image (S1005), and on thebasis of the extracted keyword, searches for similar radiographicinterpretation information from the stored keywords (S1006). This makesit possible to effectively narrow down the images to be searched for asimilar image.

The CPU 11 calculates a wavelet feature-based spatial distance betweenthe acquired image and each of a plurality of images corresponding tothe radiographic interpretation information found (S1007). The CPU 11selects one of the images corresponding to the radiographicinterpretation information found (S1008), and determines whether thespatial distance calculated for the selected image is shorter than apredetermined value (S1009).

If the CPU 11 determines that it is shorter than the predetermined value(YES in S1009), the CPU 11 determines that the image is similar, andoutputs the image as a search result (S1010). At this time, the CPU 11outputs the images that have been determined to be similar, in ascendingorder of their spatial distance. For example, the image(s) is/are outputto the display device 23 for display.

If the CPU 11 determines that the spatial distance calculated is notshorter than the predetermined value (NO in S1009), the CPU 11determines that the image is not similar, in which case S1010 isskipped. The CPU 11 determines whether all the images have been selected(S1011). If the CPU 11 determines that there is an image yet to beselected (NO in S1011), the CPU 11 selects a next image (S1012). Theprocess then returns to S1009, and the above-described processing isrepeated. If the CPU 11 determines that all the images have beenselected (YES in S1011), the CPU 11 terminates the processing.

As described above, according to the present embodiment, the waveletfeatures indicating the features of an acquired medical image can beused to search for a similar image from among the images stored as pastcases. Even an inexperienced doctor is able to find an image thatcorresponds to the most similar case and, thus, to select appropriatemedical treatment.

It is noted that the present invention is not restricted to theabove-described embodiment; a variety of modifications and improvementsare possible within the scope of the present invention.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

While the invention has been described with reference to exemplaryembodiments, it will be understood by those skilled in the art thatvarious changes may be made and equivalents may be substituted forelements thereof without departing from the scope of the invention. Inaddition, many modifications may be made to adapt a particular system,device or component thereof to the teachings of the invention withoutdeparting from the essential scope thereof. Therefore, it is intendedthat the invention not be limited to the particular embodimentsdisclosed for carrying out this invention, but that the invention willinclude all embodiments falling within the scope of the appended claims.Moreover, the use of the terms first, second, etc. do not denote anyorder or importance, but rather the terms first, second, etc. are usedto distinguish one element from another.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiments were chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A medical image search apparatus for searchingfor an image similar to a newly taken image from among medical imagesbased on past cases, comprising: a feature calculation unit configuredto calculate wavelet features of a plurality of images that have beentaken and stored in the past; a keyword extraction unit configured toextract a keyword included in radiographic interpretation informationfor each stored image; an information storage unit configured to storethe calculated wavelet features and the extracted keywords inassociation with the respective stored images; an image acquisition unitconfigured to acquire a newly taken image; a wavelet feature calculationunit configured to calculate a wavelet feature of the acquired image; aradiographic interpretation information search unit configured toextract a keyword included in radiographic interpretation informationcorresponding to the acquired image and, on the basis of the extractedkeyword, search for similar radiographic interpretation information fromthe stored keywords; a spatial distance calculation unit configured tocalculate a wavelet feature-based spatial distance between the acquiredimage and each of images corresponding to the radiographicinterpretation information found; and an output unit configured tooutput, as a search result, any images for which the calculated waveletfeature-based spatial distance is shorter than a predetermined value, inascending order of the calculated wavelet feature-based spatialdistance.
 2. The apparatus according to claim 1, wherein the featurecalculation unit and the wavelet feature calculation unit each calculatetwo-dimensional Gabor wavelet features as the wavelet features.
 3. Theapparatus according to claim 1, further comprising: a frequencydistribution vector calculation unit configured to calculate frequencydistribution vectors for all images, by calculating M said waveletfeatures for each image and binarize the respective wavelet features forconversion into an M-dimensional bit string, wherein the spatialdistance calculation unit calculates the calculated waveletfeature-based spatial distance as an angle between the calculatedfrequency distribution vectors, and wherein M is a natural number of 2or greater.
 4. A computer program product, comprising: acomputer-readable storage device; and program code embodied on thecomputer-readable storage device, wherein the program code, whenexecuted by a processor, configures the processor to: calculate waveletfeatures of a plurality of images that have been taken and stored in thepast; extract a keyword included in radiographic interpretationinformation for each stored image; store the calculated wavelet featuresand the extracted keywords in association with the respective storedimages; acquire a newly taken image; calculate a wavelet feature of theacquired image; extract a keyword included in radiographicinterpretation information corresponding to the acquired image and, onthe basis of the extracted keyword, search for similar radiographicinterpretation information from the stored keywords; calculate a waveletfeature-based spatial distance between the acquired image and each ofimages corresponding to the radiographic interpretation informationfound; and output, as a search result, any images for which thecalculated wavelet feature-based spatial distance is shorter than apredetermined value, in ascending order of the calculated waveletfeature-based spatial distance.
 5. The computer program of claim 4,wherein the wavelet features are two-dimensional Gabor wavelet features.6. The computer program of claim 4, wherein the program code, whenexecuted by the processor, further causes the processor to: calculatefrequency distribution vectors for all images by calculating M saidwavelet features for each image and binarize the respective waveletfeatures for conversion into an M-dimensional bit string, wherein M is anatural number of 2 or greater, and calculate the calculated waveletfeature-based spatial distance as an angle between the calculatedfrequency distribution vectors.